Multiple camera microscope imaging with patterned illumination

ABSTRACT

An array of more than one digital micro-camera, along with the use of patterned illumination and a digital post-processing operation, jointly create a multi-camera patterned illumination (MCPI) microscope. Each micro-camera includes its own unique lens system and detector. The field-over-view of each micro-camera unit at least partially overlaps with the field-of-view of one or more other micro-camera units within the array. The entire field-of-view of a sample of interest is imaged by the entire array of micro-cameras in a single snapshot. In addition, the MCPI system uses patterned optical illumination to improve its effective resolution. The MCPI system captures one or more images as the patterned optical illumination changes its distribution across space and/or angle at the sample. Then, the MCPI system digitally combines the acquired image sequence using a unique post-processing algorithm.

TECHNICAL FIELD

This invention relates to a microscope system that reconstructs imagesusing multiple cameras, patterned illumination and computationalpost-processing.

BACKGROUND ART

Current microscopes exhibit a tradeoff between their resolution andfield-of-view (FOV). To image across a larger FOV, one is typicallyforced to design a microscope objective lens that offers a poorerresolution. Due to this tradeoff, most standard microscopes are onlyable to capture a maximum of 50 million resolvable spots (i.e., 50megapixels) per image. This invention relates to a microscope thatovercomes the above limit by simultaneously offering a high resolutionover a large FOV.

There are a number of current devices and methods that also attempt toaddress the resolution versus FOV tradeoff in optical microscopes. Themost common strategy is to use a standard microscope objective lens,selected for a particular resolution, along with a mechanicaltranslation stage to sequentially shift the sample through the limitedFOV of the objective lens over time. This type of device is often usedin whole slide imaging. A well-known example is the Aperio system fromLeica, which can image a 15×15 mm FOV at approximately 0.65 μmresolution (i.e., comparable to a 20× objective) in one minute (one ofthe fastest on the market currently) [Ref NPL1]. An example used inoptical metrology is the ODIN microscope from HSEB-Dresden, which canimage a 300 mm diameter wafer at ˜1 μm resolution in slightly less thana minute [Ref NPL2].

To speed up the image acquisition process, one can instead increase thedesign complexity of a single objective lens to increase the lens FOVfor a fixed resolution goal. This is often the case in lithography,where such wide FOV, high-resolution lenses can contain over 30 elementsand cost several million dollars or more (see for example patentreference [Ref PTL1] from Cal Zeiss, or [Ref PTL2] from Nikon, or any ofthe other related large patented lens designs). These lithography lensesare able to maintain an NA of 0.8 (or above) over a FOV up to 100 mmacross, but often occupy up to a cubic meter of space and must bedesigned and mounted in a customized manner.

To relax the size, weight, complexity and cost of such large lenses,previous work has added an adaptive scanning mirror behind a lesscomplicated lens system, which multiplexes a larger FOV onto a singlesensor by imaging over time [Ref NPL3]. Alternatively, one can shift toa multi-lens design, the simplest being the use of multiple standardobjective lenses in parallel [Ref NPL4]. However, this type of approachdoes not attempt to image a contiguous FOV, but instead just multiplenarrow segments of the sample of interest, as with other alternativedesigns [Ref PTL3 Ref PTL6].

The same principle has also been created in a miniaturized form, usingarrays of microlenses for fluorescence imaging [Ref NPL5, Ref NPL6], aswell as with an array of Fresnel zone plates [Ref NPL7]. An entire FOVhas been imaged onto a single detector using an array of 800 largemicrolenses, which offers a significant speedup for whole-slide imaging[Ref NPL8]. However, the FOV of the arrangement in [Ref NPL8] remainslimited to the size of the utilized digital detector at the focal plane,which is still relatively small (no more than 100 mm across, and oftenless with smaller pixels). A unique property of the present micro-cameraarray invention described here is that it can scale to an arbitrarilywide FOV by simply adding more camera modules, and utilize arbitrarilysmall (and inexpensive) focal plane detector arrays. Finally, cameraarray designs have been used to acquire light field datasets within amicroscope [Ref NPL9]. However, such approaches offer a resolution andFOV that is still limited to that defined by the main objective lens.

The present invention, termed a multi-camera patterned illumination(MCPI) microscope, can capture a contiguous wide-FOV image withmicroscope-scale resolution in a single snapshot. This is the “standard”mode of MCPI operation. In addition, the MCPI system may improve theresolution performance of each camera module by shining patternedillumination onto the sample. This is the “high-resolution” mode of MCPIoperation. There are many microscopes that illuminate a sample withpatterned illumination while capturing one or more images. Examplesinclude both dark-field and structured illumination microscopes.However, no designs besides the MCPI microscope use multiplemicro-cameras to capture light from the same sample area (i.e., captureoverlapping FOV's) while at the same time providing patternedillumination.

Previous work has also considered how to improve the resolution of amicroscope by capturing more than one image [Ref NPL10]. However, thisprior work operates within a standard microscope using a singleobjective lens while capturing n different images over time. For each ofthe images, a unique set of one or more LEDs are illuminated within anLED array, which can either be located above or below the sample ofinterest. Similar prior work also reconstructed a 3D image of a thicksample using n uniquely illuminated microscope images [Ref NPL11], orsimultaneously provides patterned illumination from more than oncesource [Ref NPL12, NPL13].

Recently, a technique has been proposed to improve resolution using anarray of cameras [Ref NPL14]. With n cameras in the array, this designcan in principle simultaneously capture n unique regions of the samplespectrum (i.e., the Fourier transform of the sample's amplitude andphase transmittance, if the sample is thin). However, these regions willnot overlap in the Fourier domain, as required by certain methods forsuccessful algorithm convergence [Ref NPL10]. Furthermore, the prior artis designed for far field macroscopic imaging (of objects multiplemeters to kilometers away), is not amenable to a configuration formicroscope imaging, requires highly coherent resolution (e.g. from alaser and not from LEDs), and also does not attempt 3D imagereconstruction. A second recent experiment has attempted to use an arrayof lenses and sensors and a short working distance [Ref NPL15] toimprove resolution. However, this work also considers the case of ahighly coherent source that would spread the spectrum evenly across allof the cameras in the array, and this design does not consider cameraswith overlapping FOVs. A third recent experiment [REF US Patent App.2016503a] uses multiple cameras and LED illumination to improve theresolution of images of each well within a well plate, but this priorart considers each camera and well individually. It does not attempt toimage a contiguous FOV, and is thus not applicable to generallyunaligned samples. Furthermore, this prior art does not usemicro-cameras that offer overlapping FOVs, and thus cannot combine theimage data from more than one camera to improve the resolution at onesample plane location, as is achieved in the current invention.

There are also other methods to use patterned illumination andcomputational post-processing to improve fluorescent image resolution,as for example outlined in [Ref NPL16]. However, to the best of ourknowledge, this prior work has only been demonstrated in optical systemsthat contain a single microscope objective lens and capture a singleimage FOV. It has not been performed in a multiple camera system likethis invention.

SUMMARY OF INVENTION Technical Problem

As noted above, most current microscope platforms rely on a singlemicroscope objective lens for image formation and are often in a largeform-factor to accommodate a human viewer. Due to presence of lensaberrations, it is challenging to design a single microscope objectivelens to simultaneously offer high resolution over a wide FOV. Forexample, most commercially available objective lenses with an NA of 0.8(approximately 600 nm resolution, defined as the cutoff periodicity ofan grating imaged with λ=500 nm coherent illumination) have a 0.5 mmdiameter FOV (e.g., a 40× objective lens). Likewise, objective lenseswith a FOV of approximately 1 cm offer a cutoff resolution ofapproximately 5 μm (e.g., a 2× objective lens), which is around a 10lower resolution than the 0.5 mm FOV lens.

As the effect of lens aberrations increase proportional to the lensdiameter and the image FOV, additional optical surfaces are typicallyadded to a given lens system to offer some optical correction within thedesign [Ref NPL17]. However, this comes at the expense of lens systemsize, cost, and complexity. Instead of scaling the size of a singlelens, an alternative strategy is to use multiple lenses, which are eachattached to their own unique sensor at the focal plane (e.g., an arrayof micro-cameras). Given M micro-cameras within an array, the width ofeach camera lens can be M times smaller than a single lens offering thesame FOV, and thus the effect of aberrations within each micro-camerawill be M times less. The micro-camera array imaging strategy has beeninvestigated in the past for macroscopic imaging [Ref NPL18 Ref NPL20]as well as a unique multiscale architecture for imaging objects locatednear infinity [Ref NPL21, Ref NPL22]. However, it is typicallychallenging to reach high resolutions with a micro-camera arrayarrangement, due to the geometric constraint that the magnification ofeach micro-camera placed in array cannot exceed unity.

Solution to Problem

The current MCPI invention uses an array of micro-cameras and apatterned illumination source for microscopic imaging. Its opticaldetection system contains more than one micro-camera, positioned in anarray, to image partially overlapping FOVs of a sample. In the standardMCPI mode, one image from each micro-camera may be combined together tocreate a large FOV image at standard resolution. The MCPI microscopealso contains a source of patterned illumination, which may shine lighton the sample from a plurality of angles and/or in a particular spatialpattern, such that the spatial-angular distribution of light reachingthe sample changes over time. In the “high-resolution MCPI mode, eachmicro-camera may then acquire a unique image for each illuminationpattern. A post-processing algorithm may then combine the acquired setof images from any or all of the micro-cameras and for any or all of theillumination patterns into a high-resolution image reconstruction of thesample. The high-resolution reconstruction may also offer a measure ofsample depth, spectral (i.e., color) properties, and/or the opticalphase at the sample plane.

Advantageous Effects of Invention

The MCPI imaging system may achieve an image resolution of approximately3-15 μm in a single snapshot (i.e., in “standard” mode after eachmicro-camera acquires one image). The FOV of the MCPI system growslinearly with the number of micro-cameras included in the array. Forexample, if the FOV of one camera is 1.25×1.25 cm, then an approximately10×10 cm FOV is possible with 8×8=64 micro-cameras, and a 20×20 cm FOVis possible with 16×16=256 micro-cameras. In one preferred embodiment,the FOV of each micro-camera in the array at least partially overlapswith one or more other micro-cameras. With this overlap, it is possibleto determine the height profile (i.e., distance along the optical axis)of a sample of interest using standard stereoscopic imaging methods.

In addition, in the high-resolution mode of operation, each micro-camerain the presented system may acquire more than one image, where apatterned illumination source changes the angle and/or spatialdistribution of illuminating light at the sample plane between eachcaptured image. In one preferred embodiment, we show how it is possibleto improve the resolution of the final image reconstruction beyond thatdefined by the diffraction limit of its imaging lenses, by up to afactor of 5 or more in either dimension (e.g. from 5 μm to 1 μm or less)using a patterned illumination and post-processing strategy. In a secondpreferred embodiment, the MCPI system can improve the final imagereconstruction beyond that defined by the diffraction limit of theimaging lenses and also measure the height profile of the sample at amultitude of spatial locations. In a third preferred embodiment, MCPIsystem can improve the final image reconstruction beyond that defined bythe diffraction limit of the imaging lenses and also measure the opticalphase of the sample. In a fourth preferred embodiment, the MCPI systemcan also measure and remove the aberrations within the imaging lenses[Ref NPL23], and/or measure the spectral (i.e., color) properties of asample.

Finally, the MCPI system also offers a size/weight/complexity/costadvantage with respect to standard microscopes. It does not require anymoving parts, its micro-cameras fit within a compact space, it does notrequire a rigid support structure and can thus operate within a small,confined space.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a top-and-side view of an embodiment of the presentmicro-camera patterned illumination microscope invention.

FIG. 2 is a second top-and-side view of an embodiment of the presentmicro-camera patterned illumination microscope invention.

FIG. 3 is a component diagram of an embodiment of the presentmicro-camera patterned illumination microscope invention.

FIG. 4 is a cross-sectional side view of 3 example micro-camera unitswithin the present micro-camera patterned illumination microscopeinvention

FIG. 5 is a cross-sectional side view of the present micro-camerapatterned illumination microscope invention with example light raysshown.

FIG. 6 is a component diagram of the field-of-view (FOV) captured by oneembodiment of the present micro-camera patterned illumination microscopeinvention.

FIG. 7 is a cross-sectional side view of the present micro-camerapatterned illumination microscope invention with example light raysshown.

FIG. 8 is a look-up table (LUT) of an embodiment of the presentmicro-camera patterned illumination microscope invention.

FIG. 9 is a component diagram of the image formation process for oneembodiment of the present micro-camera patterned illumination microscopeinvention.

FIG. 10 is a table of a preferred embodiment of the present micro-camerapatterned illumination microscope invention showing example arrayparameters.

FIG. 11 is a table of a preferred embodiment of the present micro-camerapatterned illumination microscope invention showing example micro-cameraunit parameters.

DESCRIPTION OF EMBODIMENTS

General arrangement of the MCPI microscope: A diagram of one preferredembodiment of an MCPI microscope is shown in FIG. 1, which includes asketch of the micro-camera array [100], an example semi-transparentsample [200] and a patterned illumination source [300]. Here, thepatterned illumination source is an array of more than one light sources[320] positioned beneath the sample. The light from a group of lightsources in [320] can be turned on for standard MCPI mode, and this groupcan be scanned/varied while acquiring a sequence of images in thehigh-resolution operation mode. We discuss details of the preferredembodiment of each micro-camera and the patterned illumination sourcesin later sections.

FIG. 2 shows a second preferred embodiment of an MCPI microscope thatcan use both transmission illumination [300] and reflection illumination[310] (dual illumination). Here, the patterned illumination source maybe located both below and above a sample, and more than one LED canilluminate the sample from both below [320] and above [310]. The dualillumination geometry will work best with samples that are bothpartially reflective and transparent, as in certain types ofsemiconductor wafer [210]. Both proposed geometries may follow the sameprinciples and processing steps as outlined next for MCPI imaging, so wecan consider them as effectively the same, apart from the exact locationof the patterned illumination source with respect to the sample alongthe optical axis.

The general workflow of the MCPI setup is shown in FIG. 3. It should beread in general terms and not for a specific geometry of hardware, as itis well known that micro-cameras, processors, control units and computerprocessor elements can be collocated physically or can be distributedand these geometric relationships are dynamic as production volumes,costs and chip design feature sizes change over time. When light fromthe patterned illumination source in [300] reaches and interacts withthe sample of interest (i.e., can be absorbed or scattered from thesample, for example) [200], then the light exiting the sample carriesinformation about the sample to the micro-camera array [100]. Themicro-camera array contains more than one micro-camera [110]. Foroperation in transmission mode, the micro-cameras may be mounted in aperiodic rectangular array. For the proposed invention, the mount caneither be flat with respect to the sample surface, or curved into ahemispherical shape as it is in [Ref. NPL21].

After each micro-camera captures a digital image, the image data canthen be passed to a set of electronic micro-camera (MC) control units[500], which may provide logic and local memory for each micro-camera.It is common knowledge that the processor of each control unit may beembedded on the same chip as a digital detector, or may be included as aseparate chip or circuit. Each MC unit can then pass the image data to acomputer processor [600], which can contain a display [610], processor[620] and a computer readable medium [630]. The computer processor mayalso control the patterned illumination source. The MCPI microscope cancapture one or more images over time. Between each captured image, thecomputer processor may change the illumination pattern created by thepatterned illumination source [180]. After capturing one or more image,the computer processor can then perform an image post-processing stepthat can create a final high resolution, wide FOV MCPI imagereconstruction. This image reconstruction may be shown on a separatedisplay [700]. With this general workflow in mind, we now presentdetails about each individual component of the MCPI device.

The MCPI patterned illumination source: The patterned illuminationsource can illuminate the sample with light from a plurality ofdirections, wavelengths and/or spatial patterns. In one preferredembodiment, the patterned illumination source may consist of an array ofLEDs positioned at different locations. For example, the patternedillumination source could take the form of an LED array like that usedin [Ref. NPL10] (32×32 LEDs, model SMD3528, center wavelength=632 nm, 4mm LED pitch, 150 μm active area diameter). Alternatively, acustom-designed array of any number of LEDs (anywhere from 1 to 1million) might be used in any sort of circular, hexagonal, random orother geometric spatial arrangement, either on a flat or curved surface.The wavelength of the light emitted by the light sources can be in therange of 200 nm 2 μm. Wavelengths outside this range are also possible.Each light source may emit the same wavelength or a different wavelengthof light.

In a second preferred embodiment, the patterned illumination source canconsist of one or more laser sources or laser diode sources, which mayremain in a stationary position or may move positions between eachcaptured image to provide different angular or spatial patterns light tothe sample. In a third preferred embodiment, one or more laser sourcesor laser diode sources may be sent through one or more optical fiberspositioned at different locations and/or angles with respect to thesample. The light from the one or more optical fibers may reach thesample at different angular or spatial arrangements. In a fourthpreferred embodiment, a spatial light modulator (SLM), wherein thespatial light modulator comprises a liquid crystal or a liquid crystalon silicon display for displaying an illumination pattern, may be usedas the patterned illumination source. By changing the patterneddisplayed on the SLM, the illumination pattern may be changed betweencaptured images. In a fifth preferred embodiment, a digital micromirrordevice may be used as the patterned illumination source, wherein one ormore miccromirrors oriented at a first angle to reflect light towardsthe sample define a particular illumination pattern, and this patternedmay be changed between captured images. We refer to this general set ofspatially distributed optical sources as the “patterned illuminationsource”.

The MCPI micro-camera: A simplified cross-sectional diagram of anexample micro-camera is marked as [110] in FIG. 4. Each micro-cameraunit may contain one or more lenses for focusing light [120], anaperture [140], as well as a radiation detector for detecting light[130].

In one preferred embodiment, the radiation detector may contain 1-20million pixels that are 0.5 μm 5 μm in size. In the diagram in FIG. 4,the one or more lenses for focusing light uses two optical elements. Inone preferred embodiment, the lens system may contain two to ten opticalelements, and may be designed to offer a 0.1-0.9 magnification at asample working distance of 3-200 mm, similar to the specifications in[Ref. NPL21]. Other example lens and camera parameters, including thevariables marked in FIG. 4, are presented in FIG. 10. Note that weexpect the coherent resolution of this setup may be approximately 1-15μm at the sample plane, which is similar range of resolutions availableby standard 0.5×-2× objective lenses.

The MCPI Micro-Camera Array:

The MCPI micro-camera array is comprised of more than one micro-camera.In one preferred embodiment, the micro-cameras may placed adjacent toone another in a planar configuration, in which case the optical axis ofall micro-cameras are parallel to one another. The MCPI micro-camerascan be arranged in either a rectangular, hexagonal, or other form ofperiodic grid across this flat plane. A simplified cross-sectionaldiagram of a micro-camera array with 3 micro-cameras in a planarconfiguration is shown in FIG. 4. In a second preferred embodiment, themicro-camera array may be curved, in which case the optical axis of eachmicro-camera is not parallel with the other micro-cameras in the array.In this configuration, the micro-cameras towards the edge of the arraymay be angled such that their lenses can capture more light from thesample (i.e., are pointed towards the center of the sample), to improvedetection efficiency.

EXAMPLES

Light from the patterned illumination source exits the sample from manyspatial locations. Some of this light may then propagate to themicro-camera array. Considering one spatial location along the sample,the light exiting this location will pass through one or moremicro-camera lenses to form one or more images. In the most generalarrangement, each micro-camera can image a distinct sample region to itsimage plane and will record the intensity of this optical field on adigital detector array (e.g., a CMOS or CCD pixel array). We also notethat the micro-cameras do not necessarily have to form an exact image(e.g., can be defocused or otherwise optically modified, e.g., as by acoded aperture). We denote the area of the sample from which light hasinteracted with, and can then enter into micro-camera number n (heredenoted as Mn), as field-of-view n (here denoted as FOVn). What makesthe MCPI camera array geometry distinct from other camera arrays usedfor microscopy is its utilization of overlapping FOVs. That is, the sameposition on the sample may appear within FOV1 (for camera M1) and FOV2(for camera M2), for example, where M1 and M2 may denote two differentmicro-cameras that are physically adjacent to one another. Suchoverlapping regions, “FOV Overlap 1-2” and “FOV Overlap 2-3”, arelabeled in FIG. 4.

We consider a simple example of how 3 micro-cameras image a sample inFIG. 5. Each micro-camera in the MCPI system can have a unique FOV andthis FOV can overlap with the FOV of one or more adjacent cameras. Letus consider the sample location denoted by the letter “B”, which iswithin FOV1 and FOV2 for cameras M1, labeled [131], and M2, labeled[132], but not within FOV3 for camera M3, labeled [133]. Light from thepatterned illumination source, here emitting from one LED j in [322],travels at angle θ_(j), in [350] to illuminate this letter “B”. Thepattern illumination may interact with the sample and spreads intomultiple directions. Here we show three possible directions as threeunique rays, [351]-[353]. The direction of ray [352] (ϕ₂) is the same asthe direction of the patterned illumination (θ_(j)=ϕ₂) and contributesto the bright-field content of the image detected by the M2 camera[132], I₂(x), after passing through micro-camera M2's lens. A differentray [353] travels from the same sample location at angle ϕ₃ towardscamera M3 [133]. However, since the sample location containing the “B”is not within FOV3, it will not enter camera M3 and will not reach itsdetector, marked as [130]. It will thus not contribute to the associatedimage, I₃(x) [133].

The last ray [351] emerges from the sample at an angle ϕ₁ with respectto the optical axis and towards camera M1. We assume ϕ₁ is less than theacceptance angle ϕ_(a) of each micro-camera (where we defineϕ_(a)=asin(NA), with NA the micro-camera numerical aperture). Since wealso assume the letter “B” is within FOV1, ray [351] will thus enterM1's lens and contribute to an image. However, let us also assume thatin this diagram the sum of the illumination angle θ_(j) and thediffracted angle ϕ₁ exceeds the lens acceptance angle:θ_(j)+ϕ_(i)>ϕ_(a). In other words, if we were to shift the LEDillumination back to normal incidence, then ray [351] would also rotateby θ_(j) and thus be traveling at an original angle θ_(j)+ϕ₁, whichwould not pass through the lens. Thus, ray [351] can contribute to thedark-field content of the M1 image in [131]. While ray [351] originatesfrom the spatial location at the sample plane as ray [352], it containsa different type of angular information. As we detail next, the MCPImicroscope can use the unique information captured by micro-cameras M1and M2 about the same sample location (the letter “B”) to improve imageresolution and detect depth.

MCPI Data Capture:

In one preferred configuration, the MCPI patterned illumination iscomprised of an LED array, and the system illuminates one LED within theLED array at a time and captures a unique image from each and everymicro-camera within the micro-camera array. If there are a total of Nmicro-cameras and J LEDs, then the MCPI system may capture and save atotal of N×J unique images.

A useful format of MCPI data is created after additionally segmentingeach captured image into V different overlapping image segments, or“patches”. Patch formation is outlined in FIG. 6 and occurs in two mainsteps. First, The full image from each micro-camera can be aligned on a“sample plane grid” with respect to all of its neighbors. The full imagefrom micro-camera 1 (M1) is in [131] as FOV1, the full image frommicro-350 camera 2 (M2) is in [132] as FOV2, and the full image frommicro-camera 3 (M3) is in [133] as FOV3. FOV1 contains the letters ‘A’and ‘B’. ‘A’ is in image patch 1, shown as [151], and ‘B’ is in imagepatch 2, shown in [152]. FOV2 contains the letters ‘B’ and ‘C’ in imagepatch 2 and image patch 3 [153], respectively. FOV3 contains ‘C’ and ‘D’in image patch 3 and image patch 4 [154], respectively.

Images with the same feature (e.g., the letter B) are aligned with asimple image registration algorithm (e.g., a mean-squares least fit withrespect to position and orientation). This alignment is commonly used tocombine multiple images into on panorama image. The goal of imagealignment is to ensure that the same spatial location within each imageis assigned the same pixel value on a pixelated grid defined at thesample plane. For example here, the image from M2 in [132] is shifted tothe left (in pixel value) until the letter B overlaps with the letter Bin the image from M1 in [131]. Each pixel will receive the same sampleplane grid location for e.g. the pixel containing the upper corner ofthe letter “B”. The result of this alignment process is a compositeimage as shown in [155].

Once each image is aligned over the sample plane grid, the images maythen be split into patches. In the example in FIG. 6, we show each imagesplit into two patches: image [131] contains patch [141] and [142],image [132] contains patch [143] and [144], and image [133] containspatch [145] and [146]. In one preferred embodiment, these patches canoverlap slightly with one another by 0-10%, and each image will be splitinto 4—1000 patches. For example, if each micro-camera image contains1000×1000 pixels, a set of 10×10 patches can be formed by splitting theimage into 100×100 pixel square blocks. Here, the patches will overlapby 0%. As a second example, if each micro-camera image contains1000×1000 pixels, a set of 10×10 patches can be formed by splitting theimage into 120×120 pixel square blocks. Here, the patches will overlapby greater than 0%. The patch size can be as small as 2×2 pixels, or aslarge as the entire image (i.e., 1000×1000 pixels in this example).

After splitting the images into patches, the final data set for MCPIwill consist of V×N×J image patches. It can be helpful to store thisdata set as a multi-dimensional array M, where each image patch isindexed by 3 different variables, M(v,n,j), which denotes the vth imagepatch from the nth camera under illumination from the jth LED. Here,1≤v≤V, 1≤n≤N and 1≤j≤J. The array is shown as [800] in FIG. 7. In FIG.7, we also show the same light rays [351]-[353] that emerge from theletter “B” from FIG. 5. Now, we also show how the letter “B” can becontained within one patch, labeled xv as [221]. The collection of allrecorded images from all the micro-cameras after being split intopatches, as well as under all forms of patterned illumination, form thedataset M [800].

In addition to forming the MCPI data set M, it may also be helpful andnecessary to calibrate the MCPI system. In one preferred embodiment,MCPI system calibration can be achieved with a digitally saved look-uptable (LUT), which here we denote with the function L(v,n,j). The LUTmay also be indexed by the same three variables as the data matrixM(v,n,j). In one preferred embodiment, L(v,n,j) can store a vectordenoting the difference between the sine of two (average) angles:sin(ϕ_(n))-sin(θ_(j)), as shown within the table marked [810] in FIG. 8.Here, ϕ_(n) is the angle between the center of sample patch v and theoptical axis of the nth micro-camera, as denoted for 3 micro-cameras inFIG. 7 as [351]-[353]. Likewise, θ_(j) is the average angle ofillumination at patch v from the jth illumination pattern.

In one preferred embodiment, the jth illumination pattern can originatefrom the jth LED [322], in which case we may assume this illuminationacts as a plane wave, denoted by [350], across the small patch in FIG.7. This saved vector value sin(ϕ_(n))-sin(θ_(j)) indicates the centralwavevector (k_(x) ^(c), k_(y) ^(c)) of the sample optical spectrum thatpasses through micro-camera Mn when the sample patch v is illuminated byLED j. Specifically, it defines the amount of shift in the spatialfrequency domain that should be applied to the spectrum of sample patchu₂(x, y) (which we will call û₂(k_(x), k_(y))) to align it to theaperture function of micro-camera Mn, which we call a(k_(x), k_(y)) andis centered at (k_(x)=0, k_(y)=0). The LUT can allow us to efficientlycompute the aperture-spectrum product û₂(k_(x)-sin(ϕ_(nx))-sin(θ_(jx)),k_(y)-sin(ϕ_(ny))-sin(θ_(jy)))*a(k_(x),k_(y)) for any or all of the Nmicro-cameras within the micro-camera array. In one preferredembodiment, the calibration LUT table can be pre-computed for a givenMCPI setup's camera-LED geometry. In a second preferred embodiment, thecalibration LUT can be measured in a pre-calibration process. Ifmultiple types of patterned illumination are used with the MCPI system,then it can be helpful to pre-compute and/or measure and thensubsequently save multiple different LUT for use by the MCPIpost-processing algorithm.

MCPI Data Post-Processing:

A component diagram of one preferred embodiment of the MCPI imagepost-processing workflow is in FIG. 9. The MCPI data matrix M and acalibration LUT L can serve as input to the MCPI post-processingworkflow. The output of the workflow is a final image reconstructionwith improved spatial resolution. In one preferred embodiment, thisoutput may also include a measurement of depth at different spatiallocations across the image. In another preferred embodiment, the outputalso includes a measurement of the optical phase at different spatiallocations across the image. In another preferred embodiment, the outputalso includes a measurement of multi-spectral content at differentspatial locations across the image.

In the first step of the workflow, image patches may be formed asdescribed in the previous section. First, the images from all of themicro-cameras are spatially aligned over a complete sample plane grid.In one preferred embodiment, spatial alignment ensures that the samesample features in each image set occupy the same pixel locations alongthe sample plane grid. Then, the sample plane grid is split into adesired number of V image patches. In FIG. 9, [928] shows an exampleimage of a full petri dish, overlaid with an example sample plane gridthat denotes how to split up the image into multiple patches (smallsquares), marked v₁, v₂, etc. The with patch is imaged by one or moremicro-camera. In FIG. 9, patch v₂, marked [929], is imaged by twomicro-cameras (e.g., is in FOV1 from micro-camera M1 and FOV2 frommicro-camera M2.

In the second workflow step, each patch can be considered one at a time.In step [930], we consider image patch v=2. Here, we see that 2micro-cameras, M1 and M2, contain patch v=2 within their FOV. We termthe collection of images associated with one patch area from onemicro-camera an “image group”. For example, to form one image group[931], we ,ay select the set of all images from micro-camera M1 from thedata matrix: M(v=2, n=1, j=1 to J). To form another image group [932]associated with micro-camera M2, we may select the images from thedataset with M(v=2, n=2, j=1 to J). For each image group, we may alsoselect the central wavevector associated with each image from the LUT instep [933]. For image group 1 we may select L(v=2, n=1, j=1 to J), andfor image group 2 we may select L(v=2, n=2, j=1 to J), for example.These two sets of values are both in the table marked [811]. Next, for aparticular image patch, we may input the associated image groups and LUTvalues into MCPI algorithm. For example, for image patch v=2, we inputM(v=2, n=1 to 2, j=1 to J) and L(v=2, n=1 to 2, j=1 to J) into the MCPIfusion algorithm (described in the next section). The output of the MCPIfusion algorithm can then be a high-resolution image of sample patchv=2, containing both its amplitude and phase content, as shown in [812],which is saved in computer memory [813]. This workflow is repeated forall image patches, as denoted by the iteration arrow in [814]. In onepreferred embodiment, this workflow can be performed in parallel for allimage patches to improve computation time. Finally, the high-resolutionoutputs for all of the image patches can then tiled together to form afinal MCPI high-resolution image as shown in [815].

MCPI fusion algorithm: The MCPI fusion algorithm may be designed to usea set of measurements in M and the LUT values in L as input. Thesemeasurements and LUT values may be associated with the patternedillumination for the vth image patch as input. In one preferredembodiment, the MCPI fusion algorithm computes a reconstruction of thevth image patch with a resolution that is higher than that defined bythe diffraction limit of its imaging lenses (e.g. from 5 μm to 1 μm orless, or from 15 μm to 8 μm or less). In a second preferred embodiment,the MCPI fusion algorithm may additionally compute a depth map of thevth image patch. In a third preferred embodiment, the MCPI fusionalgorithm can also compute the phase of the light at the sample plane.In a fourth preferred embodiment, the MCPI algorithm may also computethe multi-spectral content of the sample.

Continuing with our example for image patch v=2, the input to the MCPIfusion algorithm can be M(v=2, n=1 to 2, j=1 to J) and L(v=2, n=1 to 2,j=1 to J). Here, for example, M includes two image sets (M1 and M2) thateach contain J uniquely illuminated images. In general M can containanywhere from 2 to 1000 image sets per patch, and anywhere from 1 to10,000 uniquely illuminated images per image set. Due to their differentspatial locations with respect to the sample, each image set may containunique angular information about each sample patch within their sharedFOV. Furthermore, each image under patterned illumination may also causedifferent spatial and angular information to reach the sensor.

In general, if we describe the sample in three dimensions by a complexfunction S(x,y,z) and we assume the optical field that interacts withthe sample and the MCPI system behaves in a linear manner, then we maydescribe the process of image formation through an equation to solve forS(x,y,z). In one preferred embodiment, we may convert the data matrix Myassociated with the images collected with respect to one patch v into avector m_(v)=vec[M_(v)], which contains all pixels detected by the MCPIsystem for the vth sample patch. Here, the vec[ ] operation transformsany n-dimensional array into a vector. Furthermore, we may consider thevth patch of the sample as S_(v)(x,y,z), and then attempt to reconstructs_(v)=vec[S_(v)] using the following matrix equation that describes theMCPI image formation process:

m _(v) =|T _(v) s _(v)|² +n  (Equation 1)

Here, the absolute value squaring is due to the ability to only detectintensity with the detector at the sample plane, and n is a vector ofadditive noise. T_(v) is a “system matrix” that describes the MCPI imageformation process for the vth patch. It may be determined from thegeometry of the MCPI setup, the LUT for the vth patch L(v=2, n=1 to 2,j=1 to J), or any other type of calibration process. Using the knownvariables my and T_(v), the goal of the MCPI fusion algorithm may thenbe to determine s_(v) by solving an inverse problem. One general form ofthis inverse problem is to minimize the mean-squared error between themeasured magnitudes and an estimate of the complex-valuedhigh-resolution sample patch:

Minimize ∥√m _(v) −|T _(v) s _(v)|∥² with respect to s _(v)  (Equation2)

Another general form is to minimize a related negative log-liklihoodfunction, which is based on a Poisson noise prior. Equation 2 is a verystandard mathematical problem that can be thought of as a cost function.There are a number of algorithms available to minimize this costfunction. In one preferred embodiment of the MCPI algorithm, analternating minimization-type strategy may be adopted to solve for themissing phase of each patch to minimize Equation 2, for example usingthe Douglas-Rachford algorithm.

In a second preferred embodiment, it is possible to solve theminimization problem in Equation 2 by constructing an AugmentedLagrangian and then minimizing the Augmented Lagrangian with gradientdescent. In a third preferred embodiment, it is possible to solveEquation 2 using an iterative optimization strategy that firstdetermines the gradients of Equation 2, or the gradients and theHessians of Equation 2, and then applying a Gauss-Newton method,somewhat similar to the methods in [Ref. NPL24]. In a fourth preferredembodiment, the sample may be fluorescent and s_(v) may be a real,positive-valued function, and a minimization method similar to thoseused in structured illumination fluorescent microscopes to determine ahigh-resolution sample may be used (e.g., an algorithm similar to one ofthe minimization methods used in [Ref NPL16] may be applied).

The MCPI fusion algorithm can use any or all of these strategies toproduce an estimate of the high-resolution sample, s_(v). As shown inFIG. 9, the invention may perform the MCPI fusion algorithm in sequenceor in parallel for all image patches to produce a set of finalhigh-resolution image reconstructions, each for a separate patch. Thesereconstructions may also include an estimate of the sample height,and/or the sample phase, and or its spectral color content. Finally,these separate patches may be combined together using an image stitchingalgorithm to produce a final, large, high-resolution imagereconstruction that can be digitally saved and/or shown on a display.

INDUSTRIAL APPLICABILITY

The invention has been explained in the context of several embodimentsalready mentioned above. There are a number of commercial and industrialadvantages to the invention that have been demonstrated, including theability to image an unbounded FOV at high resolution with a compact,lightweight, and non-moving system. The invention also provides invarying embodiments additional commercial benefits like high throughput,3D images, multi-spectral analysis and dark-field images, to name a few.

While the invention was explained above with reference to theaforementioned embodiments, it is clear that the invention is notrestricted to only these embodiments, but comprises all possibleembodiments within the spirit and scope of the inventive thought and thefollowing patent claims.

CITATION LIST Patent Literature

-   [PTL1]: K. Schuster, Projection objective for microlithography, U.S.    Pat. No. 6,801,364-   [PTL2]: Nikon Corporation, Projection optical system and projection    exposure apparatus, U.S. Pat. No. 5,805,344-   [PTL3]: K. C. Johnson, “Microlens scanner for microlithography and    wide-field confocal 535 microscopy,” U.S. Pat. No. 6,133,986 (2000).-   [PTL4]: C. F. Bevis et al., “System for inspection of patterned or    unpatterned wafers and other specimen,” US Patent App.    US2004/0246476 A1-   [PTL5]: S. K. Case et al., “High speed optical inspection system    with camera array and compact, integrated illuminator,” US Patent    App. US2011/0069878 A1-   [PTL6]: E. Rosengaus et al., “System and method for inspecting    semiconductor wafers,” U.S. Pat. No. 6,020,957 (2000).-   [PTL7]: R. Horstmeyer et al, Aperture Scanning Fourier Ptychographic    Imaging. US Patent Application US20150036038 A1 (2015).-   [PTL8]: X. Ou et al., Embedded Pupil Function Recovery for Fourier    Ptychographic Imaging Devices. US Patent Application US20150160450    (2015).-   [Claim 1] [PTL9]: R. Horstmeyer et al., Variable Illumination    Fourier Ptychographic Imaging Systems, Devices, and Methods. US    Patent Application US20150054979 A1 (2015).

1. A multi-camera patterned illumination microscope comprising: aplurality of more than one micro-camera unit, which each detect one ormore images of a distinct region of a sample that partially overlapswith one or more regions of a sample imaged by other micro-camera units;one or more optical sources that create patterned optical illuminationat the sample, where the patterned optical illumination creates anoptical field at the sample with a specific spatial and angulardistribution; a computer processor configured to convert the imagesacquired by the micro-camera units into a final image reconstruction ofthe sample with a larger field-of-view than any one of the singlemicro-camera units.
 2. The multi-camera patterned illuminationmicroscope of claim 1: that also provides a measurement of the height ofthe sample at one or more spatial locations.
 3. The multi-camerapatterned illumination microscope of claim 1: where the optical fieldcreated by patterned optical illumination has a spatial and angulardistribution that varies as a function of time.
 4. The multi-camerapatterned illumination microscope of claim 1: where more than onemicro-camera units are arranged in a geometric array, with eachmicro-camera unit placed immediately adjacent to the other micro-cameraunits.
 5. The multi-camera patterned illumination microscope of claim 1:that converts the images acquired by the micro-camera units into a finalimage reconstruction of the sample with a larger field-of-view and ahigher detection sensitivity as compared to the field-of-view anddetection sensitivity of any one of the single micro-camera units. 6.The multi-camera patterned illumination microscope of claim 1 wherethere are more than one multi-camera patterned illumination microscopeslocated adjacent to at least one other multi-camera patternedillumination microscope, such that an extended microscope system isconfigured.
 7. The multi-camera patterned illumination microscope ofclaim 1: where the one or more optical sources that create patternedoptical illumination consist of approximately 1-5000 lasers or laserdiodes located at different spatial locations.
 8. The multi-camerapatterned illumination microscope of claim 1: where one or more opticalsources that create patterned optical illumination are mounted to themicro-camera array to illuminate the sample from the same side of thesample as the micro-camera array, and/or one or more optical sourcesthat create patterned optical illumination are located on the oppositeside of the sample with respect to the micro-camera array.
 9. Themulti-camera patterned illumination microscope of claim 1: where the oneor more optical sources illuminate the sample with light of differentcolors (different optical wavelengths).
 10. The multi-camera patternedillumination microscope of claim 1: where each micro-camera unit iscomprised of a lens system for focusing light, an aperture forrestricting the light that passes through the lens system, and aradiation detector that detects one or more images over time.
 11. Themulti-camera patterned illumination microscope of claim 6: where theimages from the plurality of multi-camera microscopes are digitallycombined to also provide a measure of the multi-spectral content of thesample at one or more spatial locations.
 12. The multi-camera patternedillumination microscope of claim 6: where the images from the pluralityof multi-camera patterned illumination microscopes are digitallycombined to also provide a measure of the height of the sample.
 13. Themulti-camera patterned illumination microscope of claim 1: where themicro-camera array contains approximately 10-500 micro-camera units. 14.The multi-camera patterned illumination microscope of claim 1: where thecomputer processor simultaneously reconstructs the optical phase of thesample while forming the final image reconstruction.
 15. Themulti-camera patterned illumination microscope of claim 1: where thecomputer processor also simultaneously reconstructs the height of thesample while forming the final image reconstruction.
 16. Themulti-camera patterned illumination microscope of claim 1: where thecomputer processor also simultaneously reconstructs the multispectralcontent of the sample while forming the final image reconstruction. 17.The multi-camera patterned illumination microscope of claim 1, where theone or more optical sources that create patterned optical illuminationconsist of approximately 15000 light emitting diodes (LEDs) located atdifferent spatial locations.
 18. The multi-camera patterned illuminationmicroscope of claim 6, where images from the plurality of multi-camerapatterned illumination microscopes are digitally combined to form afinal image reconstruction with a larger field-of-view and a higherdetection sensitivity as compared to the field-of-view and detectionsensitivity of any one of the single micro-camera units.
 19. Themulti-camera patterned illumination microscope of claim 1: where thefield-of-view (FOV) of each micro-camera unit overlaps approximately5%-90% with the FOV of one or more micro-camera units that areimmediately adjacent to it in the micro-camera array.
 20. Themulti-camera patterned illumination microscope of claim 1: where acalibration look-up-table (LUT) is stored in the computer processor toassist with the formation of the final image reconstruction. 21-24.(canceled)